An Investigation of Tracing Overheads on High End Systems

Although event tracing of parallel applications offers highly detailed performance information, tracing on current leading edge systems may lead to unacceptable perturbation of the target program and unmanageably large trace files. High end systems of the near future promise even greater scalability challenges. In this work we identify and quantify the overheads of application tracing. We report results for two ASC Purple Benchmarks with different communication characteristics: SMG2000, which exhibits an extremely high message rate, and SPhot, an embarrassingly parallel application with relatively little communication. We investigate several different sources of overhead related to tracing: instrumentation, differing trace buffer sizes, periodic buffer flushes to disk, system changes, and increasing numbers of processors in the target application. Our results show that tracing overhead is affected by differences in system software, as well as the choice of trace buffer size. As expected, the overhead of instrumentation correlates strongly with the number of events; however, our results indicate that the overhead of writing the trace buffer increases with increasing numbers of processors.